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ABSTRACT 

This paper presents a mixed-method evaluation design 
which utilizes concept mapping along with questionnaire and 
interviews* Recent developments in mixed-method evaluation have 
helped identify five purposes which mixed-method evaluation designs 
tend to serve* These are triangulat ion, complementarity, development, 
initiation, and expansion* The present paper discusses how the 
inclusion of concept mapping in designing and implementing an 
evaluation project can effectively strengthen the findings in line 
with each of these purposes. The design is presented with the help of 
three illustrative examples of evaluation and planning studies. A 
critical analysis of the purposes served by each of the components in 
these studies is presented within the framework of mixed-method 
design. This includes discussions on the characteristics of the 
methods, the importance of the sequence in which they are employed, 
and the variation in characteristics of how the methods are 
juxtaposed for each study. Three tables and four figures are 
included. (Contains 12 references*) (Author) 
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Mixed-Method Design 

Recent literature on Mixed Method evaluation designs have called for a broader 
basis for mixing of methods (Greene, Caracelli, & Graham, 1989). Earlier literature on 
the topic called for the use of multiple methods as a means f or enhancing the validity of 
inference from a study (Campbell & Fiske, 1959; Webb, Campbell, Schwartz, & 
Sechrest, 1966). In seeking to enhance validity of a study researchers employed two 
methods for the ostensible purpose of measuring the same construct. This trend has 
started to change with the call for critical multiplism (Cook, 1985; Mark & Shetland, 
1987) and an attempt to combine methods as well as paradigms (Greene et al.). As a 
result, more and more evaluation studies are relying upon mixing quantitative and 
qualitative methods not just to enhance the validity of findings but to enhance the overall 
scope of a study. This is reflected in the review of 57 studies using mixed-method designs 
by Greene et al. 

Greene et al. (1989) do an excellent job of classifying the studies according to five 
characteristic purposes for which mixed-method designs were used. These are: 1) 
Triangulation, 2) complementarity, 3) Development, 4) Initiation, and 5) Expansion. Of 
these, triangulation is the most common purpose for which mixed-method designs have 
been used previously. Through triangulation one seeks to enhance the validity of research 
findings by corroboration, convergence, or correspondence of results from different 
methods. In other words, different methods are employed to measure the same construct 
so that same results obtained from different methods would be harder to refute. 
Complementarity of methods occurs when the results from one method helps to enhance 
or clarify the results from another method. In development, the results from one method 
helps inform or develop the other methods. Discovery of paradox or contradiction, 
gaining new perspectives, and recasting of questions or results from one method with 
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questions or results from another method distinguish studies where initiation has been the 
purpose of using mixed-method designs. Finally, when different methods are used for 
different inquiry components to extend the breadth and range of inquiry, the purpose is 
termed as expansion. In trying to present a mixed-method design with concept mapping, 
questionnaire and interviews, Greene et al.'s paper is used here as the framework because 
of its extensive discussion of purpose and characteristics of mixed-method designs. The 
purposes may not always seem mutually exclusive nor should they be considered 
exhaustive. However, the intent of this paper being the presentation of a practical mixed- 
method design, laying the foundation for a modified theoretical framework remains 
beyond its scope. 

A central feature of mixed-method designs is the combination of a qualitative and 
a quantitative method (Kidder & Fine, 1987; Greene et al., 1989). A typical example of 
such a design would be the combination of a questionnaire or a scale and interviews 
representing the quantitative and qualitative components respectively. Deviations from 
such a typical design are common too. A questionnaire may contain open-ended 
questions, the interviews may be highly structured and so on. Such deviations may 
highlight one of the components more than the other. However, the basic intention of 
expanding the scope of the design would remain. Such variations will be evident from the 
research designs presented in this paper. 

Apart from the characteristics of the methods themselves the other important 
features that need to be considered are the sequential order in which the different 
components of the design are implemented and how each of the purposes listed in Greene 
et al. (1989) is served. The order in which the different methods are implemented has a 
direct bearing on the purpose of mixed-method designs. Some of the purposes are not 
served unless the methods are implemented either in sequence or concurrently. For 
instance, most of the studies reviewed by Green et al. reveal that sequential 
implementation was common among those seeking development or initiation. Concurrent 
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implementation on the other hand was more common among studies seeking 
complementarity. 

Concept Mapping 

In addition to the questionnaire and interview components of mixed-method 

designs, this paper proposes the inclusion of concept mapping in mixed-method designs 

as a third component to further enhance the scope of a study. Trochim (1989, p. 1) 

provides the following definition of concept mapping: 

Concept mapping is a type of structured conceptualization which 
can be used by groups to develop a conceptual framework which 
can guide evaluation or planning. 

The use of concept mapping can address several issues in mixed-method designs 
raised by Greene et al. ( 1989). The term 'development of a conceptual framework' used in 
the definition by Trochim ( 1989) can be specifically stated to include purposes such as 
informing the other methods (development), recasting of questions or gaining new 
perspectives (initiation), and expansion of breadth and range of inquiry (expansion). It is 
difficult, however, to characterize concept mapping as a strictly quantitative or a 
qualitative method. This issue has been raised by Trochim (1989) in the form of whether 
concept mapping is a soft science or a hard art. He discusses the difficulty of classifying it 
either as an exact science or as what is generally understood as an art form. He discusses 
the commonalties that concept mapping has with both science and art, and suggests that 
the dichotomy may remain and be accepted as such. In a similar vein, one may argue that 
the dichotomous nature of concept mapping, i.e., qualitative vfs-a-vfs quantitative, is hard 
to resolve. A look at the steps required for concept mapping explains this. 

The process of concept mapping involves six steps (Trochim, 1989). These are: 

1. Preparation including selection of participants and development of focus 
for the conceptualization. 

2, Generation of statements. 
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3. Structuring of statements. 

4. Representation of statements in the form of a concept map using 
multidimensional scaling and cluster analysis. 

5. Interpretation of maps. 

6. Utilization of maps. 

Of these steps the fourth one is solely quantitative in nature and utilizes 
multivariate statistical methods to generate the actual concept maps. The third step has 
some elements of quantitative methods in that it requires the participants to rate and sort 
the statements generated in the second step according to their importance and similarity 
respectively. The other four steps are qualitative in nature. Each of these steps relies upon 
the qualitative judgment of the various stakeholders. For instance, the development of a 
focus statement may utilize the judgment of the administer ve personnel, the generation 
of statements and interpretation of maps require the judgment of the participants, and 
finally, how the maps are to be utilized for evaluation purposes is determined by the 
participants in conjunction with the evaluator through discussions or negotiation. 

From the preceding summary of the process of concept mapping, it is ev ident that 
the method by itself is a blend of qualitative and quantitative elements. Both elements 
play significant roles and are both required to make the process meaningful. In discussing 
the method characteristics in the following studies, concept mapping will therefore be 
characterized as mixed while the questionnaire will be characterized acco" !ing to its 
content and the interviews as qualitative. 

The rest of this paper is devoted to presenting three different studies exemplifying 
mixed-method design with concept mapping, questionnaire and interviews . The major 
thrust of tho discussion will be highlighting the methodological aspects and less attention 
will be paid to the substantive issues. Discussions on each of these studies will contain: a) 
a brief description of the study, b) characteristics of the methods employed, c) the 
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sequence of implementation of different methods, and d) the characteristics of mixing of 
methods including the purposes of such mixing. 

New York Academy of Medicine (N YAM) 

This study was commissioned by the New York Academy of Medicine to evaluate 
their ongoing "Being Healthy!" (BH) program implemented in the middle schools of New 
York city (Trochim & SenGupta, 1993). The goals of this program were two-fold: 1) To 
implement a cost-effective model of comprehensive health education in selected 
intermediate and junior high schools for grades seven through nine, and 2) to coordinate 
the project with all other health education and health services initiatives in the schools at 
both the organizational and content levels. A major component of the implementation 
process was an intensive teacher training given each year to a new batch of teachers, 
administrators, and counselors from the selected schools. In addition, the project staff 
were available for consultation throughout the school year to help with any problem or 
difficulty arising from implementation of the curriculum. 

At the time the evaluation project was undertaken the program had been running 
for five years. The goal of the project was to evaluate the effects of the Being Healthy! on 
the participating teachers 1 . In addition, the project also aimed at finding out, as much as 
possible, how the students, the parents, the community and the school administration 
were affected as a result of the implementation of "Being Healthy". Four major research 
questions were investigated in this project: 

1. Prior Training - Before participating in Being Healthy! program, what 
experiences and skills did teachers and counselors have with regard to 
comprehensive health education? 

2. Effects of Being Healthy! Training - How effective was the training 

provided by the Being Healthy! program? 

term teacher will indicate the participants in general who include counselors and administrators in 
addition to teachers unless otherwise noted. 
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3. Effects of Program Participation - How has participating in the Being 
Healthy! program affected teachers' knowledge (e.g., about various aspects 
of health), attitudes (e.g., toward teaching, morale) and behavior (e.g., 
participation in their school, professional development, personal 
development)? 

4. Observed Effects on Students - How has participating in Being Healthy! 
affected students (knowledge, attitude, behavior) from the perspective of 
the teachers? 

A mixed-method approach was adopted to address the guiding research questions. 
In this particular study, where only a certain segment of the participants, namely, the 
teachers were available for eliciting information, the use of mixed-method design with as 
many sources of information as possible was necessary to enhance the scope of research. 
The stated methodological goals in this study were: 

1. To contribute to the development of measures used in later stages of the 
evaluation (development). 

2. To strengthen the findings of particular methods (triangulation). 

3. To get a richer picture of a particular issue as explored in different stages 
of the evaluation through different methods (expansion). 

4. To enhance and illustrate the findings of one method with the results of 
other methods (complementarity). 

5. To allow for comparison with previously collected data. 

The mixed method design consisted of four components. A summary of the 
methods and their characteristics are presented in Table 1. The participants in the "Being 
Healthy!" program formed the sampling frame for the study. The total number of such 
participants was two hundred and six. The questionnaire was sent to all the participants 
and the response rate was fifty per cent. A subsample of twelve participants were 
interviewed over the phone. The concept mapping session was conducted with sixteen 
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participants. The sequence in which the methods were implemented is summarized in 
Figure 1. 



Method 


Description 


Characteristics 


OiiPQti nnnairp 

V^liVOllUllllvlll V 


A paper and pencil instrument consisting of 
both closed and open-ended questions. 


Mixed 


Interview 


Extensive open-ended interviews with a 
subsample of teachers. 


Qualitative 


Concept Mapping 


A one day session with a subsample of 
teachers. 


Mixed 


Existing Database 


Analysis of relevant parts of the existing 
database comprising of data routinely 
collected by the New York Academy of 
Medicine from the BH participants. 


Quantitative 



Table 1 - Description and characteristics of methods used: NY AM 



Purt of the questionnaire addressed the same research questions that the interviews 
did. As a result, the analyses of interviews and questionnaire allowed for triangulation of 
findings. It may be noted that the main criterion for triangulation, i.e., different methods 
measuring the same phenomenon was fulfilled by the use of two equally important 
methods implemented independently. The purpose of complementarity was also served 
by the use of both questionnaire and interviews. Together, they provided the means to 
probe into different aspects of the research questions and many of the findings from the 
questionnaire f which had a much larger number of respondents) were enhanced and 
illustrated very well by the interviews in keeping with the stated purpose of 
complementarity. Expansion occurred through the emergence of a much richer picture of 



Questionnaire 
development & 
administration 



Concept 




Telephone 


Mapping 




interviews 



Simultaneous analysis of 
questionnaire, interview 
and existing database 



Figure 1 - Sequence of Implementation: NY AM 
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the evaluand as a result of using multiple methods (including the analysis of existing 
database) at different stages of the evaluation. Finally, the use of concept maps in 
analyzing the interview data as well as the responses to the open-ended questions served 
the purpose of development. The categories obtained in concept mapping provided the 
initial framework or coding scheme for the qualitative data. Statements obtained from the 
interviews and the questionnaire were categorized according to the concept mapping 
categories; new categories were started as and when a statement did not fit any of the 
existing categories. Not only was the analysis of the qualitative data more informed as a 
result of using the concept mapping categories, but w.j coding was made considerably 
easier for these data. 

Tompkins Community Hospital (TCH) 

This study was commissioned by the Tompkins Community Hospital - a county 
hospital in upstate New York, to evaluate their Clinical Technician program (Trochim, 
1992). Like many other hospitals around the country TCH was faced with a growing 
shortage of nursing professionals during the 1980's. To meet this problem TCH embarked 
on an in-house training program to develop a second line of caregivers, termed the 
Clinical Technicians (CT), as support staff to the Registered Nurses (RN). The training 
was tailor-made for the hospital's needs and intended mostly for people who were already 
working in the hospital in some related positions. The aim was to maintain the same level 
and quality of care in the face of growing nursing shortage. In the first phase of the 
program the CTs were all employed in the Intensive Care Unit (ICU) where the ratio of 
the number of nurses to that of the patients was necessarily the highest and consequently 
the nursing shortage was the most acutely felt. At the time of the evaluation the second 
phase of the program had just started and CTs had been introduced in the Emergency 
Room. 
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The evaluation was undertaken by a group of students from Cornell University as 
part of a two semester-long evaluation practicum course. The goal of this evaluation was 
to determine how the productivity and employee satisfaction in the intensive care unit 
were affected by the Clinical Technician program. The major research questions 
investigated were: 

a) Productivity - 

1 . Has there been a change in ICU productivity since the implementation of 
the CT program? If so, has there been an increase or a decrease in 
productivity? 

2. What the RNs' and CTs' perceptions of productivity? 

b) Employee Satisfaction - 

L What are the overall levels of job satisfaction and stress among RNs and 
CTs in the ICU? Is there a correlation between stress levels and job 
satisfaction? Does partnership with a CT increase or decrease a) job 
satisfaction, b) stress levels of RN? 

2. What are the staff issues regarding the CT program? 

3. What is the impact of the CT program on recruitment, retention and 
absenteeism of ICU staff? 

4. Are there any significant unexpected effects and features of the CT 
program? 

The overall research design for this study was a mixed-method one. However, 
within the sub-areas of productivity and employee satisfaction, the changes were 
measured using quasi -experimental designs (Cook & Campbell, 1979). The other aspects 
of the study such as, the perception and issues related to CT program were addressed 
through interviews and concept mapping. The stated methodological goals were similar to 
those of the NYAM study. The four components of the research design are presented in 
Table 2. 
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Method Description Characteristics 



Questionnaire 


A paper and pencil instrument consisting of 


Quantitative 


Interview 


Semi-structured interviews with a subsaniple 
of nursing staff. 


Qualitative 


Concept Mapping 


Two sessions with a subsample of nursing 
staff. 


Mixed 


Secondary Analysis 
of Data 


Analysis of relevant parts of the existing 
database maintained by TCH. These 
included indices of productivity, 
absenteeism, retention and recruitment data. 


Quantitative 



Table 2 - Description and characteristics of methods used: TCH 



The sampling frame for the study consisted of the nursing staff (both RNs and 
CTs) in the Intensive Care Unit. The questionnaire was given to all the RNs and CTs and 
the response rate was one hundred percent. Such high rate of response can be ascribed to 
the close nature of the study setting. Again all the nursing staff were interviewed and all 
of them participated in the concept mapping sessions along with the nursing administrator 
of the ICJ. The sequence of implementation of different methods is presented in Figure 
1. 

Simultaneous analysis of 
questionnaire, interview 
and existing database 

Fi gurc 2 - Sequence of implementation: TCH 

In this study, concept mapping played a key role in the conceptualization process. 
The categories obtained from concept mapping were utilized both in developing the 
questionnaire and structuring the interviews. While the questionnaire contained two 
sections on productivity and employee satisfaction, a third section consisted solely of 




Questionnaire 
development & 
administration 



Interviews 
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items which emerged from concept mapping. Similarly, some of the questions asked 
during the interviews were also based on the issues raised through concept mapping. The 
use of concept mapping results in such manner can lead to initiation as fresh perspectives 
may emerge through concept mapping and those can be probed further through interviews 
and questionnaire. This can happen in addition to serving the more obvious purpose of 
development when concept mapping is used in developing the other methods. 

Descriptive analysis through the interviews complemented the analysis of the 
questionnaire response and the secondary analysis in this study. This study also provides 
a typical example of expansion as interviews were employed to assess the program 
processes while the quantitative methods were intended to find out about the program 
outcomes in terms of productivity. A combination of two quasi-experimental designs, 
viz., time series and non-equivalent control group design was used to assess the program 
outcome. While the time series data came from the ICU records (secondary analysis), the 
data for non-equivalent control group came from the Emergency Room records. The use 
of quasi -experimental designs helped answer the validity issues in assessing productivity 
and employee satisfaction outcomes while the interviews on the same topics provided the 
means for triangulation. 



St Catherine of Siena 

The purpose of this study was strategic planning rather than evaluation. However, 
the particular mixed-method design employed in this study makes it worthwhile to 
mention in this paper as a planning counterpart to the ones mentioned so far. The St. 
Catherine of Siena Parish council embarked on a self-study strategic planning process to 
articulate its mission and long-term goals (Trochim et al., 1991). The overall planning 
process needed a total of six steps for completion. The second and third steps of this 
process which involved data collection and analysis respectively and were carried out by 
a parish appointed data gathering committee, are highlighted here. The data gathering 
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committee consisted of four volunteer parishioners in addition to five undergraduate 
students from Cornell University who participated under a program called the Faculty- 
Fellows-in-Service. 

The data gathering committee was responsible only for data collection, analysis 
and formulating appropriate methodology to carry out such tasks. The committee decided 
on a mixed-method research design consisting of four components - concept mapping, 
focus groups, interviews, and a survey (questionnaire). The method characteristics are 
presented in Table 3. 



Method 


Description 


Characteristics 


Questionnaire 


A paper and pencil instrument consisting of 
both open and closed-ended questions. 


Mixed 


Interview 


Semi -structured interviews with a subsample 
of "key respondents". 


Qualitative 


Concept Mapping 


Conducted with the parish council members 


Mixed 


Focus Groups 


Conducted with the parishioners 


Qualitative 



Table 3 - Description and characteristics of methods used: St. Catherine of Siena 

The concept mapping session was attended by the parish council members while 
the focus groups were attended by the parishioners. "Key respondents" for the interviews 
came from St. Catherine staff members, parish council members, parishioners, local 
community leaders, leaders of other faith groups, and leaders of nearby Catholic 
communities. The survey questionnaire was distributed among all parish-registered 
household and spouses. The response rate to the survey was twenty nine percent. The 



1 Concept 




Focus 


1 Mapping 




Groups 



Interviews 



Survey 



Data Analysis p . 
Integration of 
Results 



Figure 3 - Sequence of Implementation: St. Catherine of Siena 
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sequence of implementation of different components of the study is shown in Figure 3. 

In this study, concept mapping was used to develop the other three methods - 
focus groups, questionnaire for the survey, and structuring of the interviews. The analyses 
of data obtained through each of these methods were also guided by the concept mapping 
categories. While the interviewers and facilitators were responsible for summarizing the 
interviews and the issues that came up during the focus groups, thematic analyses of these 
summaries were based on the concept mapping categories. Similarly, the results from the 
survey were also categorized in terms of the major concept mapping themes. The use of 
different methods helped expand the breadth of the study and allowed a broad range of 
participants to highlight the different aspects of parish functioning. This increased the 
usefulness of the study findings in terms of developing a mission statement and strategic 
planning for the future. 

Conclusion 

The preceding discussion of three practical applications of mixed-method designs 
gives an idea of how concept mapping can be combined with questionnaire and 
interviews to strengthen mixed-method designs. Addition of existing database or focus 
groups can add a further dimension to the research findings. It may be noted that there are 
possibilities of several variations on a basic structure consisting of concept mapping, 
questionnaire and interviews. However, there are two basic variations that are apparent 
from the studies presented here. Both of these depend on how concept mapping is 
integrated into the research design. 

The first of these variations is exemplified by the NYAM research design. In this 
study the development of other methods was independent of concept mapping. As a 
result, the position of concept mapping in the sequence of implementation was immaterial 
so long as it happened before the analysis and integration stage. Concept mapping helped 
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in this study in analysis and integration of results. The an t ysis was both informed and 
better organized by the use concept mapping. In this design, therefore, initiation if it 
occurs, can occur in a narrow sense, viz., recasting the results from one method with 
results from another method (Greene et al., 1989). Development can also occur in rather a 
narrow sense by virtue of other methods being better informed at the analysis stage by the 
results of concept mapping. 

The manner in which concept m< ^ping is used in the other two studies contrasts 
with the first variation in that concept mapping is used as a vehicle to lay a foundation to 
the study design. The intent of development is more clearly pronounced as is that of 
initiation. This is demonstrated very well in the St. Catherine study and to a lesser degree 
in the TCH study. In both these designs concept mapping's position in the sequence of 
implementation had to be in the first place. The research design in the St. Catherine study 
made all the other methods dependent on concept mapping from development through 
analysis and integration of results. This was in keeping with the planning intent of the 
study where the whole process started with a clean slate and it was important to obtain an 
insight into what the stakeholders were looking for. The evaluation intent of the TCH 
study necessitated a different approach in both development of and analysis of the results 
from the other methods. There was a clear focus in terms of the basic constructs that 
needed to be evaluated as well as the luxury of an existing database which allowed quasi- 
experimentai designs to be functional within the purview of an overall mixed-method 
design. The developments of the questionnaire and the interview schedules were 
principally guided by the evaluation questions and well supplemented by the concept 
mapping categories. 

A suggested mixed-method design involving concept mapping, questionnaire and 
interviews is presented in Figure 4. A fourth component, viz., previous data has been 
deliberately added to represent the studies presented in this paper. The fourth component 
is optional and if included, can be varied according to the needs of a particular study. 
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Concept 
Mapping 



Questionnaire 




Interviews 



Previous 
Data 



Figure 4 - Schematic diagram of a mixed-method design involving concept 
mapping, questionnaire and interviews. 

Similarly, the nature of the questionnaire and the interviews is not fixed. The 
characteristics of the questionnaire especially, can vary from totally quantitative to 
completely qualitative. The diagram in Figure 4 also points out how the different 
purposes listed by Greene et al. ( 1989) can be served by this design. Again, the purpose 
of any given study will depend on its methodological goal ; however, this design may 
prove suitable for any study with an emphasis on one of these purposes. 
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ABSTRACT 



This paper presents a mixed-method evaluation design which utilizes concept 
mapping along with questionnaire and interviews. Recent developments in mixed-method 
evaluation have helped identify five purposes which mixed-method evaluation designs tend 
to serve. These are triangulation, complementarity, development, initiation, and expansion. 
The present paper discusses how the inclusion of concept mapping in designing and 
implementing an evaluation project can effectively strengthen the findings in line with each 
of these purposes. The design is presented with the help of three illustrative examples of 
evaluation and planning studies. A critical analysis of the purposevS served by each of the 
components in these studies is presented within the framework of mixed-method design. 
This includes discussions on the characteristics of the methods, the importance of the 
sequence in which they are employed, and the variation in characteristics of how the 
methods are juxtaposed for each study. 
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